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Background / Context: 

Description of prior research and its intellectual context. 

The criteria for determining the student outcomes that define a school as having “turned 
around” are not well defined (Kutash, Nico, Gorin, Rahmatullah, & Tallant, 2010), and the 
definition of turnaround perfonnance varies across studies (Aladjem, Binnan, Orland, Harr- 
Robins, Heredia et ah, 2010; Hennan, Dawson, Dee, Greene, Maynard et ah, 2008). Although 
current policy initiatives offer guidelines for identifying CLP schools, there is no standard 
definition or methodology in common usage. The Federal School Improvement Grants (SIG) and 
Race to the Top (RTT) initiatives, for example, require states to identify the lowest five percent 
of schools but allow states flexibility to set critical parameters such as the period over which 
perfonnance should be measured. Similarly, federal policy sets expectations that states will 
measure school improvement progress against benchmarks but allows states to establish the 
specific benchmarks (within parameters) and methods of measuring progress. 

School-based accountability measures, which have dominated the relatively small 
literature on empirically identifying low-perfonning and turnaround schools, are problematic on 
three points: 

1. Percentage measures (e.g., percent proficiency, graduation rates, etc.) ignore 
important variations in tested outcomes that occur on either side of the outpoint. 

2. School-based measures ignore the implied measurement error when calculating 
across schools of differing sizes. 

3. School-based measures ignore compositional changes in the student body from cohort 
to cohort (Kane & Staiger, 2002). 

One may reasonably expect these problems to be exacerbated in the case of identifying 
low-perfonning and turnaround schools. Attempting to identify low performers by definition 
focuses on schools with perfonnance at the extreme low end of the distribution; these are the 
same schools that are most likely to be subject to large corrections over time if the observed 
perfonnance was due to measurement error. Therefore, distinguishing authentic low performance 
and authentic improvements from corrections due to random fluctuation in enor-prone measures 
is the primary challenge of attempts to empirically identify this phenomenon. 

A second critical issue in identifying CLP and TA schools is that accountability measures 
are generally status measures, which confound pre-existing differences among students with 
differences in school quality. Growth measures are generally regarded as better measures of 
isolating schooling inputs (Meyer, 1997; Raudenbush, 2004). Status measures are much more 
stable over time (slower to show signs of improvement), while growth measures are less 
correlated over time (Kane & Staiger, 2002; Linn & Haug, 2002). The implication for TALPS is 
that a focus on status measures would underidentify TA schools or identify improvements only 
several years after the turnaround actually occurred. A focus on both status and growth would be 
more likely to capture movement toward improved outcomes. Kane and Staiger (2002) contend 
that using student-level growth measures can overcome the criticisms of school-based 
accountability measures raised above: growth measures are generally calculated using student- 
level data along continuous measures, can be adjusted according to the imprecision in the 
estimate, and capture within-student improvements rather than differences between cohorts. 
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Purpose / Objective / Research Question / Focus of Study: 

Description of the focus of the research. 

The primary research question for this substudy is as follows: What proportion of CLP 
schools display specific performance trajectories over time? In particular, what proportion of 
CLP schools display: quick, dramatic improvement (TA schools); weak net improvement (MI 
schools); or negatively sloped performance trajectories (NI schools)? 

Setting: 

Description of the research location. 

(May not be applicable for research methods or panel submissions) 

Administrative data from three states on student test scores were used (Florida, North 
Carolina, and Texas), from which two separate data samples were created in each state for 
elementary and middle schools. The data spanned six school years (2002-02 through 2007-08). 

Population / Participants / Subjects: 

Description of the participants in the study: who, how many, key features, or characteristics. 

(May not be applicable for research methods submissions) 

We make three restrictions to the universe of schools for inclusion in the study. First, to 
ensure that the school structure is consistent with our estimation approach, we require that a 
school report test outcomes for all the grades of interest (grades 3-5 for elementary schools, 
grades 6-8 for middle schools). Second, to ensure that we have sufficient data over a long period 
on which to judge improvements in perfonnance, we require that a school serve all these grades 
for all six school years in the analysis time span. Finally, to ensure that schools are internally 
stable over time, we require that each school have at least 50 percent of its students re-enroll in 
the following year for each of the five observed year-to-year transitions in the data. Table 1 
presents descriptive statistics of the universe of data and the schools used in the Florida 
elementary school sample, one of six data samples used in the study. 

<Insert Table 1 Here> 

Intervention / Program / Practice: 

Description of the intervention, program, or practice, including details of administration and duration. 

(May not be applicable for research methods submissions) 

Not applicable. 

Research Design: 

Description of the research design. 

Through a series of simulations and iterative data analysis using the data samples from 
three states, we develop a set of rules to empirically identify CLP schools, and then classify 
schools as TA/MI/NI based on subsequent performance. The model we employ to estimate pre- 
and post-period perfonnance in schools is a three-level hierarchical linear model, in which 
student’s test scores over time are nested within students, which are in turn nested in schools. 

The model for a representative elementary school sample is described below. 
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Level- 1 (within-student): 

Y tik 7t()/A + K | //. T i mcl 7I 2 /aT i IUc2//a + 8 fifc 

£-tik ~ 

V(0, a 2 ) 

(1) 

Level-2 (between-student; within-school): 
none = Pooa + PoiA Post_statU8y* + r 0 ,A 

%A ~ 

V(0, x^o) 

(2a) 

K\ik = PlOA + pilAPost_g34/A 



(2b) 

n 2i k = P 20 A + p2iAPost_g45 //c 



(2c) 

Level-3 (between-school): 
Poo a = 9ooo + Vooa 

Vooa 

~N( 0, xoo) 

(3a) 

PoiA = 0010 + V 01A 

Voia 

~N( 0, xoi) 

(3b) 

PlOA - 0100 + V 10A 

V 10A 

~V(0,Xio) 

(3c) 

P 1 1 A = 0110 + V ha 

V 1 !A 

~N( 0,Tn) 

(3d) 

p 20 A = 0200 + V 20 A 

V20A 

~N(0, x 20 ) 

(3e) 

p 21 A = 0210 + V 21 A 

V 21 A 

~N( 0, X 21 ) 

(3f) 


This model allows both status and growth parameters for both pre- and post-periods in all 
schools to be estimated simultaneously. Specifically, in Equation 1, Timely took values -1 for 
grade 3, 0 for grade 4, and 0 for grade 5. Time2 rt<r was coded -1 for grade 3, -1 for grade 4, and 0 
for grade 5. In this coding scheme, noijk, niijk, and 712 p represent, respectively, status at grade 5, 
growth between grades 3 and 4, and growth between grades 4 and 5. In Equations 2a, b, and c, 
each predictor was coded 0 for outcomes in the pre-period and 1 for those in the post-period. 
Note that Post status^ is 1 if the cohort was in 5th grade in the post-turnaround period. 
Post_g34 //c is 1 if the cohort was in 4th grade in the post-turnaround period. Post_g45 7 A is 
identical to Post_status/A within cohort, because 5th grade achievement and the grade 4-5 gain 
are realized simultaneously. Note that we constrain growth between grades to be constant for all 
students within a school (during either the pre- or post-period), to make the model tractable; 
though we allow the intercept to vary across students within schools. This model was applied 
separately for each cut point. 

Schools were classified as chronically low performing based on the estimated random- 
effect parameters Vooa, Vio/t, and V 2 oa, which represent a given school’s average pre -period 
status, growth from grade 3 to 4, and growth from grade 4 to 5, respectively. If a school’s 
estimate of status, Vooa, fell in the lowest 15 percent of all school estimates and its estimated 
growth over the two periods, (Vioa + V20A), fell in the lowest 40 percent, the school was 
considered a chronically low-perfonning school. 

For a school to be considered a TA, the post-period status (Voia) and growth parameters 
(V 1 1 * + V 21 A) had to show statistically significant increases during the post-period, in addition to 
meeting the policy-relevant thresholds of both a 5 percentile-point improvement in the school’s 
percentile in the post-period for status and ranking at or above the 65th percentile of all schools’ 
growth in the post-period. If either of the parameters (status or growth) were statistically 
significant but did not meet the policy-relevant thresholds, the school was categorized as a MI 
school. Schools that exhibited no statistically significant improvement in either parameter 
estimate were labeled NI schools. 

Statistical, Measurement, or Econometric Model: 

Description of the proposed new methods or novel applications of existing methods. 
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(May not be applicable to submissions in sections other than research methods) 


Not applicable. 

Usefulness / Applicability of Method: 

Demonstration of the usefulness of the proposed methods using hypothetical or real data. 

(May not be applicable submissions in sections other than research methods) 

The methods developed in this study are broadly applicable, and are helpful for states in 
prioritizing intervention efforts to turnaround low-performing schools, and in recognizing signs 
of improvement among CLP schools. 

Data Collection and Analysis: 

Description of the methods for collecting and analyzing data. 

(May not be applicable for research methods submissions) 

No supplemental data collection. 

Findings / Results: 

Description of the main findings with specific details. 

(May not be applicable for research methods submissions) 

Overall findings for the paper are summarized in Table 2. Supplementary evidence 
comparing performance trajectories of CLP schools by TA, MI, and NI status are presented in 
Figures 1, 2, and 3; these and other investigations suggest the identification method developed 
here is valid and viable. 

<Insert Table 2, Figures 1, 2, and 3 Here> 

Conclusions: 

Description of conclusions, recommendations, and limitations based on findings. 

This paper summarizes the lessons learned from this exercise of empirically identifying 
CLP schools and binning them into performance categories based on their trajectories. This 
paper provides guidance for others charged with a similar task. Specifically, we learned the 
critical importance of using student-level data (rather than school-level aggregate measures), 
using growth-based measures in conjunction with status-based perfonnance metrics, the stability 
of these performance metrics over time, and how to empirically recognize turnaround in schools 
as it occurs. 

Also, we learned that low-performing schools turned around their perfonnance more 
frequently than one might have presumed based on prior research. In Florida, we identified 
approximately 15% of chronically low-performing elementary and 14% of chronically low- 
perfonning middle schools as turnarounds. Similar rates were observed in North Carolina — 13% 
and 16%, respectively; and even higher in Texas — 29% and 31%, respectively. 
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Table 1. Descriptive Statistics of FL Elementary Schools 


Descriptive Statistics of FL E 

lementary Schools 


2003 Universe 
of Data 

2007 Universe 
of Data 

2007 Sample 

Demographic Variables 

Percentage female 

48.7% 

48.5% 

48.6% 

Percentage White 

49.5% 

44.5% 

44.4% 

Percentage LEP 

8.0% 

7.9% 

7.8% 

Percentage ever eligible for 
free/reduced price lunch 

62.0% 

63.5% 

64.0% 

Institutional Variables 

Percentage charter 

2.1% 

3.8% 

2.4% 

Percentage K-5 

86.8% 

85.5% 

88.1% 

Percentage K-8 

2.4% 

3.4% 

2.5% 

Percentage serving 3-5 with alternative 
structure 

9.7% 

9.9% 

9.4% 

Number of schools with forward 
percentage <50% 

41* 

37** 

0 

Number of schools that close in the 
following year 

43 

80 

0 

Number of unique students in grades 3, 
4, or 5 

574,125 

565,596 

490,237 

Total number of schools 

1,906 

2,148 

1,599 

*out of 1750 schools (serving all of grades 3-5 at minimum in 2003 and 2004) 

**out of 1938 schools (serving all of grades 3-5 at minimum in 2007 and 2008) 

Note: a scnooi witn an alternative structure " serves an grades b-b, out is not K-b or K.-8, 
and may potentially serve grades outside the 3-5 range. The forward percentage is 
calculated as the number of students in grades 4 and 5 in school S in year t who were in 
grades 3 and 4 in S in year t-1 out of the total number of students in grades 3 and 4 in S 
in year t-1. 


Table 2. Summary Results 


Final Count of CLP, 

, TA, Ml, 

and Nl Schools by State 


Total Schools 

CLP Schools 

TA 

Ml 

Nl 

| Florida j 

Elementary 

1,599 

224 

34 

91 

99 

Middle 

535 

57 

8 

28 

21 

Total 

2,134 

281 

42 

119 

120 

North Carolina 

Elementary 

1,095 

154 

20 

48 

86 

Middle 

504 

80 

13 

29 

38 

Total 

1,599 

234 

33 

77 

124 

| Texas j 

Elementary 

2,662 

466 

136 

220 

110 

Middle 

1,023 

121 

38 

65 

18 

Total 

3,685 

587 

174 

285 

128 

| Three-state Total [ 

Elementary 

5,356 

844 

190 

359 

295 

Middle 

2,062 

258 

59 

122 

77 

Total 

7,418 

1,102 

249 

481 

372 


Note: Schools that are CLP in both reading and math are separately 
assigned to TA, Ml, and Nl status for each subject (e.g., a school 
could be TA in reading and Ml in math). In this table, a TA school is 
TA in either reading or mathematics, an Ml school is neither TA nor 
Nl in reading or mathematics, and an Nl school is Nl in either reading 
or mathematics and never TA in reading or mathematics. 
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Figure 1. 


Cohort Trajectory in Average TA School 
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Figure 2. 

Cohort Trajectory in Average Ml School 
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Figure 3. 


Cohort Trajectory in Average Nl School 
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